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(54) Image sensing apparatus. 

@ In an image sensing apparatus of this embo- 
diment a variable angle prism for preventing a 
camera shake is used to attain a pixel shift 
Consequently, low resolution images free from 
a blur and high-resolution images obtained by 
the pixel shift can be selectively obtained with 
an extremely simple arrangement Whether a 
resolution increasing function is to be executed 
is properly determined in accordance with 
photographing conditions. Therefore, the resol- 
ution increasing function has no adverse effect 
on other functions. In addition, a power-saving 
effect can be enhanced since an unnecessary 
resolution increasing function is not performed. 
Since the resolution can be further increased by 
using a conventional camera shake preventing 
device, a larger amount of a high-frequency 
component can be detected. Consequently, 
focus detection can be performed accurately. 
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BACKGROUND OF THE INVENTION 

The present invention relates to an image sens- 
ing apparatus and, more particularly, to a high- 
resolution image sensing apparatus using an image 
sensor such as a charge coupled device (CCD). 

A recent advance of image sensors such as a 
charge coupled device (CCD) has achieved a high im- 
age quality, a low price, and a small size of image 
sensing apparatuses, e.g., household compact video 
cameras and still video cameras. Consequently, a 
satisfactory image quality can be obtained by a tele- 
vision standard of, e.g., NTSC. In addition, camera 
shake correction is currently performed in video cam- 
eras and the like, and this further improves the image 
quality of dynamic images. 

A camera shake is a phenomenon in which pho- 
tographed images move vertically or . horizontally 
while a user is performing photographing by holding 
a video camera in his or her hands, since the hands 
or the body of the user slightly moves independently 
of the user's intention. Images thus photographed 
give a viewer a considerable uncomfortableness 
when reproduced on a television monitor or the like. 

To avoid this camera shake phenomenon, con- 
ventional video cameras make use of, e.g., a variable 
angle prism (to be referred to as a "VAP" hereinafter). 

A practical example of an arrangement of a con- 
ventional image sensing apparatus including a VAP 
for camera shake correction will be described below 
with reference to Fig. 15. 

In Fig. 15, a VAP 104 is constituted by coupling 
two glass plates 104a and 104b via a bellows-HIike 
spring member 104c and sealing an optically trans- 
parent liquid 1 04d in the space surrounded by the two 
glass plates 104a and 104b and the spring member 
104c. Shafts 104e and 104f provided in the glass 
plates 104a and 104b are connected to an actuator 
1 03 for horizontal driving and an actuator 1 08 for vert- 
ical driving, respectively. Therefore, the glass plate 
104a is rotated horizontally, and the glass plate 104b 
is rotated vertically. 

Note that the VAP 104 is described in Japanese 
Patent Laid-Open No. 2-12518 and so a detailed de- 
scription thereof will be omitted. 

A horizontal angular velocity sensor 101 detects 
an angular velocity caused by a horizontal motion of 
the image sensing apparatus resulting from a camera 
shake or the like. A control unit 102 performs an ar- 
ithmetic operation for the detection signal from the 
angular velocity sensor 101 such that this horizontal 
motion of the image sensing apparatus is corrected, 
and detects and supplies an acceleration component 
to the actuator 1 03. This actuator 1 03 drives the glass 
plate 104a of the VAP 104 horizontally. 

The rotational angle of the glass plate 104a which 
can be horizontally rotated by the actuator 103 is de- 
tected by an angle sensor 105. The control unit 102 



performs an arithmetic operation for this detected ro- 
tational angle and supplies the result to the actuator 
103. 

A vertical angular velocity sensor 106 detects an 
5 angular velocity caused by a vertical motion of the im- 
age sensing apparatus resulting from a camera 
shake or the like. A control unit 107 performs an ar- 
ithmetic operation for the detection signal from the 
angular velocity sensor 106 such that this vertical mo- 
10 tion of the image sensing apparatus is corrected, and 
detects and supplies an acceleration component to 
the actuator 108. This actuator 108 drives the glass 
plate 104b of the VAP 104 vertically. 

The rotational angle of the glass plate 1 04b which 
15 can be vertically rotated by the actuator 108 is detect- 
ed^ an angle sensor 109. The control unit 107 per- 
forms an arithmetic operation for this detected rota- 
tional angle and supplies the result to the actuator 
108. 

20 An image sensing optical system 110 forms an 

image of an object to be photographed on an image 
sensor 111. This image sensor 111 is constituted by, 
e.g., a CCD. A two dimensional solid state CCD is 
used in conventional image sensing apparatuses 

25 such as video cameras. An output from the image 
sensor 111 is output to a recording apparatus or a tel- 
evision monitor through a signal processing circuit 
(not shown). 

In the conventional image sensing apparatus 

30 with the above arrangement, the horizontal and vert- 
ical angular velocities caused by a camera shake are 
detected. On the basis of the angular velocities de- 
tected, the actuators move the VAP horizontally and 
vertically to refract incident light, thereby performing 

35 control such that the image of an object to be photo- 
graphed does not move on the image sensing plane 
of the image sensor. Consequently, the camera 
shake is corrected. 

On the other hand, with recent spread of image 

40 sensing apparatuses, an increasing demand has aris- 
en for a higher image quality. To meet this demand, 
it is being attempted to increase the number of pixels 
of an image sensor such as a CCD in an image sens- 
ing apparatus or to develop a high-speed I/O appara- 

45 tus or an image information compressing/decoding 
apparatus. 

For example, the number of pixels of a conven- 
tional two-dimensional solid state CCD is normally 
400,000, and the number of pixels of even a high- 

50 resolution CCD is at most 2,000,000. These pixel 
numbers currently available are unsatisfactory to ob- 
tain a sufficient resolution necessary for images, 
hard copies, and computer graphics to be displayed 
on large screens. 

55 The following methods, therefore, are disclosed 
as a method of realizing a high image quality with a 
limited number of pixels of an image sensor. 

(1) A method in which an image of an object to be 
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photographed is divided by a prism and photo- 
graphed by a plurality of image sensors, as found 
in some presently available video camera recor- 
ders. 

(2) A method as proposed in, e.g., Japanese Pa- s 
tent Laid-Open No. 60-250789, in which an im- 
age region of a photographing optical system is 
split by, e.g., a secondary imaging optical sys- 
tem, and the individual split regions are photo- 
graphed by a plurality of image sensors and then 10 
synthesized. 

(3) A method of pixel shift proposed in, e.g., Jap- 
anese Patent Publication Nos. 50-13052, 59- 
18909, and 59-43035. In this method, an element 

for splitting a light beam, such as a dichroic mirror 15 
or a half mirror, is arranged on the image plane 
side of an image sensing optical system, and a 
plurality of area sensors are arranged to be shift- 
ed from each other by an amount corresponding 
to a half pixel pitch or smaller. In this method it is 20 
possible to obtain, from a plurality of resulting im- 
ages, information containing pixels in a number 
larger than the number of pixels of the area sen- 
sors. Also, "Highvision 2/3-inch compact CCD 
camera using dual green method" in Television 25 
Society Magazine, "Image Information Engineer- 
ing and Broadcasting Technology", Vol. 47, No. 2, 
1993 has reported a method in which two of three 
CCDs for receiving separated light beams from a 
trichromatic separation prism are arranged such 30 
that their pixels are shifted from each other. In 
this method an object to be photographed is pho- 
tographed with this pixel-shifted arrangement 

(4) Japanese Patent Publication No. 57-31701 or 
64-863 has discloseda method by which a pixel 35 
shift is performed by moving the position of inci- 
dent light to an image sensor relative to the image 
sensor in synchronism with the read timing of an 
output electrical signal from the sensor. This is 
accomplished by, e.g., inserting a birefringence 40 
polarizing element into a photographing optical 
system and controlling the element By periodi- 
cally photographing the resulting images, it is 
possible to obtain information containing pixels in 

a number larger than the number of pixels of the 45 
image sensor. Research and development of this 
pixel shifting method have been made extensive- 
ly to date. 

As discussed above, the camera shake prevent- 
ing function and the resolution increasing function so 
have been developed in conventional image sensing 
apparatuses. Unfortunately, in these conventional 
image sensing apparatuses the camera shake pre- 
venting function and the resolution increasing func- 
tion are independent of each other. Therefore, the 55 
conventional image sensing apparatuses must have 
independent arrangements to include both of the two 
functions. An image sensing apparatus including 



both these functions, therefore, is difficult to manu- 
facture at a low cost and miniaturize. 

In addition, in a conventional image sensing ap- 
paratus which realizes a high-resolution by using the 
pixel shifting method as described above, this pixel 
shifting function and various other automatic adjust- 
ing functions of the image sensing apparatus may in- 
terfere with each other, thereby causing a function 
failure. As an example, if a pixel shift is performed by 
slightly vibrating a focusing lens group or an image 
sensor in the optical axis direction while an automatic 
focusing device is operating using TV signal AF (Auto 
Focus), no stable peak value output of a luminance 
signal can be obtained. This degrades the original 
performance of the automatic focusing device. Also, 
t if a pixel shift is done while another automatic adjust- 
ing device is in operation, out-of-focus images or over 
exposed-images result in some cases. In addition, if 
a pixel shift is performed during zooming, the moving 
amount of the position of incident light to an image 
sensor cannot be held constant Consequently, it may 
become impossible to obtain a satisfactory image 
quality. 

Furthermore, since a pixel shift is done although 
no satisfactory effect can be expected, the resolution 
increasing function operates unnecessarily, consum- 
ing an extra power. This is especially a serious prob- 
lem in portable image sensing apparatuses in which 
power is supplied by a battery. 

SUMMARY OF THE INVENTION 

It is an object of the present invention to provide 
an image sensing apparatus including both of a cam- 
era shake preventing function and a resolution in- 
creasing function with a simple arrangement 

According to the present invention, the foregoing 
object is attained by providing an image sensing ap- 
paratus comprising image forming means for forming 
an image of an object to be photographed, image 
sensing means for photographing the image of the 
object formed by the image forming means, displac- 
ing means for displacing the image of the object 
formed by the image forming means and the image 
sensing means relative to each other, and control 
means for performing switching between a first im- 
age sensing mode in which the displacing means per- 
forms a vibration correction to yield a low resolution 
image and a second image sensing mode in which 
the displacing means performs a pixel shift to yield a 
high-resolution image. 

. In accordance with the present invention as de- 
scribed above, the first or second image sensing 
mode is selected either arbitrarily by a user or auto- 
matically. Consequently, when dynamic images are 
required, information of dynamic images free from a 
blur can be obtained by moving the relative positions 
of an image of an object to be photographed, which 
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is formed by an optical system, and an image sensor 
by an amount by which a camera shake is corrected. 
When a high-resolution is necessary, the relative 
positions of the image of the object formed by the opt- 
ical system and the image sensor are finely moved to 5 
photograph a plurality of images, and these images 
are synthesized. 

It is another object of the present invention to pro- 
" vide an image sensing apparatus capable of properly 
determining whether a resolution increasing function 10 
is to be executed, thereby enhancing a power-saving 
effect. 

According to the present invention, the foregoing 
object is achieved by providing an image sensing ap- 
paratus comprising image forming means for forming 15 
an image of an object to be photographed, image 
sensing means for photographing the image of the 
object formed by the image forming means, displac- 
ing means for displacing the image of the object 
formed by the image forming means and the image 20 
sensing means relative to each other, synthesizing 
means for obtaining a high-resolution image by syn- 
thesizing a plurality of images of the object obtained 
by performing a pixel shift by the displacing means, 
changing means for changing photographing condi- 25 
tions, and control means for determining, in accor- 
dance with the photographing conditions changed by 
the changing means, whether the displacing means is 
to be driven. 

In accordance with the present invention as de- 30 
scribed above, driving of the displacing means for 
performing a pixel shift can be controlled in accor- 
dance with the photographing conditions. Conse- 
quently, an unnecessary pixel shift is not performed. 

It is another object of the present invention to pro- 35 
vide an image sensing apparatus capable of further 
increasing the resolution by using an image sensor 
with a limited number of pixels. 

According to the present invention, the foregoing 
object is attained by providing an image sensing ap- 40 
paratus comprising image forming means for forming 
an image of an object to be photographed, image 
sensing means for photographing the image of the 
object formed by the image forming means, displac- 
ing means for displacing the image of the object 45 
formed by the image forming means and the image 
sensing means relative to each other, synthesizing 
means for obtaining a high-resolution image by syn- 
thesizing a plurality of images of the object obtained 
by performing a pixel shift by the displacing means, so 
and vibration correcting means for selectively per- 
forming a vibration correction by using the displacing 
means. 

In accordance with the present invention as de- 
scribed above, the resolution can be further in- 55 
creased in addition to being able to perform a vibra- 
tion correction. 

The invention is particularly advantageous since 



there can be provided an image sensing apparatus by 
which an image recording mode is selected either ar- 
bitrarily by a user or automatically, and so it is possi- 
ble to selectively obtain low resolution images free 
from a blur caused by a vibration such as a camera 
shake and high-resolution images. Since these two 
types of images are formed by using a common ar- 
rangement, the configuration of the image sensing 
apparatus is exceedingly simplified. 

In addition, whether the resolution increasing 
function is to be executed is properly determined. 
Consequently the power-saving effect is enhanced, 
and the resolution increasing function is prevented 
from having an adverse influence on other operations 
of the apparatus. 

> Furthermore, since a high-resolution can be real- 
ized by using a conventional camera shake prevent- 
ing device, a larger amount of a high-frequency com- 
ponent can be detected. This makes an accurate fo- 
cus detection feasible. 

Other features and advantages of the present in- 
vention will be apparent from the following descrip- 
tion taken in conjunction with the accompanying 
drawings, in which like reference characters desig- 
nate the same or similar parts throughout the figures 
thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The accompanying drawings, which are incorpo- 
rated in and constitute a part of the specification, il- 
lustrate embodiments of the invention and, together 
with the description, serve to explain the principles of 
the invention. 

Fig. 1 is a block diagram showing the arrange- 
ment of an image sensing apparatus according to 
the first embodiment of the present invention; 
Figs. 2A and 2B are views showing the relation- 
ship between a pixel arrangement and an image, 
and the pixel arrangement of a high-resolution 
image after synthesis, respectively, in a second 
image sensing mode of the first embodiment; 
Fig. 3 is a block diagram showing the arrange- 
ment of an image sensing apparatus according to 
the second embodiment of the present invention; 
Figs. 4A and 4B are views showing the relation- 
ship between a pixel arrangement and an image, 
and the pixel arrangement of a high-resolution 
image after synthesis, respectively, in a second 
image sensing mode of the second embodiment; 
Figs. 5Aand 5B are block diagrams showing the 
arrangement of an image sensing apparatus ac- 
cording to the third embodiment of the presentin- 
vention; 

Figs. 6Ato 6C are views for explaining focus de- 
tection according to TV signal AF in the third em- 
bodiment; 

Fig. 7 is a flow chart showing pixel shift start de- 
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termination processing in the third embodiment; 
Fig. 8 is a block diagram showing the arrange- 
ment of an image sensing apparatus according to 
the fourth embodiment of the present invention; 
Fig. 9 is a timing chart of CCD driving in the fourth 5 
embodiment; 

Fig. 10 is a view showing the way the position of 
an image of an object to be photographed is shift- 
ed on an image sensor in the fourth embodiment; : 
Fig. 11 is a view showing the relationship be- 10 
tween the inclined angle of a light beam and the 
shifted amount of an image of an object In the 
fourth embodiment; 

Fig. 12 is a view showing the way an image is 
shifted in the fifth embodiment of the present in- 15 
vention; 

Fig. 1 3 is a block diagram showing the arrange- 
ment of the sixth embodiment of the present in- 
vention; 

Figs. 1 4 A to 1 4E are views showing other means 20 
for performing a pixel shift in the first to sixth em- 
bodiments; and 

Fig. 15 is a block diagram showing an illustrative 
arrangement of a conventional image sensing 
apparatus. 25 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

Preferred embodiments of the present invention 30 
will now be described in detail in accordance with the 
accompanying drawings. 

<1st Embodiment 

35 

Fig. 1 is a block diagram showing the basic ar- 
rangement of an image sensing apparatus according 
to the first embodiment of the present invention. In 
Fig. 1, the same reference numerals as in Fig. 15 
which illustrates one conventional example denote 40 
the same parts, and a detailed description thereof will 
be omitted. 

Referring to Fig. 1, reference numeral 151 de- 
notes a pixel number controller 151. In an image sen- 
sor 111, light-receiving portions of individual pixels 45 
are arranged in two directions, the X and Y directions, 
with a spacing larger than the light-receiving portion 
of one pixel. In this embodiment, the relative positions 
of an image of an object to be photographed, which 
is formed by an image sensing optical system 110, so 
and the image sensor 111 are displaced, and the pos- 
ition of a VAP 104 is changed such that the light- 
receiving portions of the individual pixels of the image 
sensor 111 apparently cover a light-insensitive por- 
tion between the light-receiving portions. The pixel 55 
number controller 151 outputs a control signal for this 
purpose to control units 102 and 107. On the basis of 
the output control signal from the pixel number con- 



troller 151, the VAP 104 is driven by actuators 103 
and 108. 

To use the image sensing apparatus of this em- 
bodiment, a user first determines the operation 
mode. Reference numerals 152 and 153 are mode 
switches for this operation. The image sensing appa- 
ratus of this embodiment has a first image sensing 
mode in which camera shake prevention is executed 
although the resolution is relatively low, and a second 
image sensing mode in which the resolution is rela- 
tively high but no camera shake prevention is per- 
formed. Each of the mode switches 152 and 153 is 
closed to a position a in the first image sensing mode 
and to a position b in the second image sensing 
mode. In the first image sensing mode, therefore, the 
VAP 104 is so controlled as to correct a blur caused 
by a vibration such as a camera shake on the basis 
of horizontal and vertical angular velocity information 
(acceleration information). In the second image sens- 
ing mode, on the other hand, the VAP 104 is so con- 
trolled that the number of pixels is seemingly in- 
creased, i.e., the light-receiving portions of the pixels 
apparently cover the light-insensitive region between 
the light-receiving portions. 

An analog signal obtained by the image sensor 
111 is converted into a digital signal by an A/D conver- 
ter 154 and applied to a scan converting unit 155 in- 
cluding frame memories 1 56a to 1 56d. In the first im- 
age sensing mode, the scan converting unit 155 does 
notf unction. That is, the digital image signal is always 
written in, e.g., the frame memory 1 56a and output in- 
tact via a data selector 157. In the second image 
sensing mode, on the other hand, the scan converting 
unit 155 stores the output digital image signal from 
the A/D converter 154 divisionally in frame memories 
1 06a to 1 06d using a method to be described in detail 
later. The data selector 157 of the scan converting 
unit 155 outputs a high-resolution image by alternate- 
ly reading the frame memories 1 56a to 1 56d. The out- 
put signal in which the number of pixels is increased 
by the scan converting unit 155 is output to a record- 
ing unit 159, a monitor display 160, or a printer 161 
through a signal processing unit 158. 

Details of the second image sensing mode for ob- 
taining high-resolution images will be described be- 
low. Figs. 2A and 2B are plan views each showing, in 
a time series manner, a relative movement between 
the light-receiving portion of the image sensor 111 
and an image of an object to be photographed in the 
image sensing apparatus of this embodiment Note 
that in Figs. 2A and 2B, it is assumed that an image 
of an object to be photographed is fixed in position. 
Fijg. 2A is a view for explaining the pixel arrangement 
, in the light-receiving portion of the image sensor 111. 
In Fig. 2A, the light-receiving portion of the image 
sensor 111 consists of NxM pixels, i.e., N pixels in the 
X direction and M pixels in the Y direction. For simpli- 
city of explanation, some of these pixels are extracted 



5 



9 



EP 0 669 757 A2 



10 



in Fig. 2k. Of the extracted pixels, a pixel of interest 
is given a pixel number, "1", and its position is indicat- 
ed by coordinates (ij). Fig. 2B shows the pixel ar- 
rangement obtained by synthesis when the data se- 
lector 1 57 alternately reads the frame memories 1 56a 
to 156d. 

Referring to Fig. 2A, in the light- receiving portion 
the pixel of interest is moved relatively in order of "1" 
-j. "2" "3" "4" for each frame of the image sensor 
111 by the VAP 104 which is driven by the actuators 
103 and 108 on the basis of the control signal from 
the pixel number controller 151. That is, the pixel of 
interest is shifted by 1/2 of the distance between in- 
dividual light-receiving portions. Assuming the 
frames corresponding to the pixel positions "1" to "4" 
in Fig. 2A are first to fourth frames, the frame period 
of an image synthesized by the data selector 157 is 
given by the sum of the first to fourth frame periods. 

in each of the first to fourth frame periods, there- 
fore, the image data from the image sensor 111 are 
sequentially stored in the corresponding frame mem- 
ory. The data thus stored are read out by the data se- 
lector 157 and finally recorded, displayed, or printed. 
An image obtained as the final output at this time is 
synthesized as illustrated in Fig. 2B. Referring to Fig. 
2B, the total number of pixels is 2Nx2M, and pixels 
photographed during the individual corresponding 
frame periods by the positions "1" to "4" of the pixel 
of interest, Fig. 2A, are present at positions (21-1,21- 
J) f (2i,2j-1), (2i-1,2j), and (2i,2j). That is, the resolution 
is doubled in both the X and Y directions. 

The above explanation is made assuming that an 
image of an object to be photographed is fixed in pos- 
ition. In practice, since an image of an object is re- 
fracted by the VAP 104, the relative position of the im- 
age with respect to the image sensor 11 1 is displaced. 

As discussed above, the image sensing appara- 
tus according to this embodiment includes the first 
image sensing mode in which camera shake preven- 
tion is performed although the resolution is low, and 
the second image sensing mode in which the resolu- 
tion is high but no camera shake prevention is done. 
Consequently, a user can selectively obtain and out- 
put high-resolution images and images free from a 
blur. 

It is also possible to output high-resolution im- 
ages thus photographed as a dynamic image by re- 
peatedly performing photographing in the second im- 
age sensing mode described above. Furthermore, 
during photographing in the first image sensing 
mode, the operation mode can be switched to the sec- 
ond image sensing mode only for one cycle from the 
first to fourth frames mentioned above. This makes it 
possible to obtain a still image with a desired high- 
resolution. 



<2nd Embodiment 

The second embodiment of the present invention 
will be described below. 

5 Fig. 3 is a block diagram showing the arrange- 

ment of an image sensing apparatus according to the 
second embodiment of the present invention. In Fig. 
3, the same reference numerals as in Fig. 15 which 
illustrate one conventional example denote the same 

10 parts, and a detailed description thereof will be omit- 
ted. 

Referring to Fig. 3, reference numeral 162 de- 
notes a motion detector for detecting the motion of an 
object to be photographed from a signal obtained by 
15 A/D converting the output from an image sensor 111. 
A motion detection method in this second embodi- 
ment can be, e.g.. the method described in Japanese 
Patent Laid-Open No. 1-98563. In this method, a plur- 
ality of images photographed at different times 
20 (fields) are compared in units of fine blocks, thereby 
detecting the motion vector of the object It is of 
course possible to employ other well-known methods 
of motion detection. 

Image sensing processing in the second embodi- 
25 ment having the above arrangement will be described 
below. As with the image sensing apparatus of the 
first embodiment discussed above, the image sens- 
ing apparatus of the second embodiment has a first 
image sensing mode for preventing a camera shake 
30 and a second image sensing mode for obtaining high- 
resolution images. To photograph an object to be pho- 
tographed by using the image sensing apparatus of 
the second embodiment, mode switches 152 and 153 
are first closed to their respective positions a to per- 
35 form photographing in the first image sensing mode. 
If the motion detector 162 determines that the object 
is moving, the image sensing apparatus immediately 
performs recording, display, or the like in the first im- 
age sensing mode as it is. 
40 On the other hand, if the motion detector 162 de- 

termines that the object is at rest, the mode switches 
152 and 153 are closed to their respective positions 
b t thereby automatically switching the mode of the im- 
age sensing apparatus to the second image sensing 
45 mode for obtaining a high-resolution. 

As described above, in the second embodiment 
the photographing mode is automatically switched 
between the first and second modes in accordance 
with the moving amount of an object to be photo- 
so graphed. This makes it possible to perform photo- 
graphing in the first image sensing mode if an object 
is moving, and in the second image sensing mode if 
an object is at rest Consequently, optimum images 
can be obtained constantly. 
55 In the above first and second embodiments, in 

the high-resolution second image sensing mode the 
number of pixels is increased at the same magnifica- 
tion in the X and Y directions of an image of an object 
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to be photographed. It is of course possible to in- 
crease the number of pixels at different magnifica- 
tions in the X and Y directions. It is also possible to 
shift an image of an object and the image sensor 111 
relative to each other only in, e.g., the X direction. 5 

In addition, in the above embodiments an image 
of an object to be photographed is displaced such that 
the individual light-receiving portions of the image 
sensor 111 apparently cover the light-insensitive re- ! 
gion between the light-receiving portions. An output 10 
received during the displacement of each light- 
receiving portion is displayed or recorded as an out- 
putfrom one pixel. This is equivalent to improving the 
resolution in both the X and Y directions. However, 
the present invention is not limited to the above em- 15 
bodiments. For example, the number of pixels can be 
increased by displacing an image of an object to be 
photographed such that the light-receiving portions 
apparently overlap each other. 

Figs. 4A and 4B are plan views each showing, in 20 
a time series manner, a relative movement between 
the light-receiving portions of the image sensor 111 
and an image of an object to be photographed. Note 
that in Figs. 4A and 4B, it is assumed that an image 
of an object to be photographed is fixed, as in Figs. 25 
2Aand 2B. In Fig. 4A, the light-receiving portions of 
the image sensor 1 1 1 are moved to seemingly overlap 
each other. Consequently, like in Figs. 2Aand 2B dis- 
cussed earlier, an image with a doubled resolution in 
both the X and Y directions can be obtained as illu- 30 
strated in Fig. 4 B. In this case, if each overlapped por- 
tion between the light-receiving portions is 1/2 of the 
light-receiving portion, MTF (Modulation Transfer 
Function) is weakened. Consequently an aliasing dis- 
tortion caused by the image sensor 111 is sup- 35 
pressed. This eliminates the need for an optical low 
pass filter consisting of a quartz crystal or the like, 
which is usually attached to the front surface of the 
image sensor 111 in order to remove any aliasing dis- 
tortion. 40 

Note that in the above first and second embodi- 
ments, in the second image sensing mode the rela- 
tive positions of an image of an object and the image 
sensor 1 11 are not changed in accordance with vibra- 
tion detection. Since the frequency of a camera shake 45 
is generally about 10 Hz, a control signal for vibration 
correction can be superposed on a control signal for 
increasing the number of pixels which is supplied by 
the pixel number controller 151. As a result, high- 
resolution images free from a blur can be obtained in so 
the second image sensing mode. 

Note also that in the first and second embodi- 
ments discussed above, the relative positions of an 
image of an object to be photographed and the light- 
receiving portions of the image sensor 111 are shifted 55 
by a 1/2 pixel pitch in the horizontal direction and/or 
the vertical direction. This pitch need only be 1/N (N 
is a natural number of 2 or larger) of the pixel pitch. 



In this case, N images of an object, of course, must 
be photographed. 

<3rd Embodiment 

The third embodiment of the present invention 
will be described below. 

In this third embodiment, an image sensing appa- 
ratus including a zoom function will be described. 
Figs. 5A and 5B show the arrangement of the image 
sensing apparatus according to the third embodi- 
ment Reference numerals 201a to 201d denote zoom 
lens groups, in which 201b denotes a variator lens 
group for varying the magnification; and 201d, a fo- 
cusing lens group. In Figs. 5A and 5B, these zoom 
lens groups are constituted by lens groups having 
positive, negative, positive, and positive refracting 
powers. However, various arrangements are possible 
as an optical system having a variable magnification 
function or a focusing function. Therefore, the pres- 
ent invention is not limited to the arrangement shown 
in Figs. 5A and 5B. 

Reference numeral 202 denotes an aperture 
stop; and 203, an interline type CCD. The quantity of 
light entering the zoom lens groups 201 is adjusted by 
the aperture stop 202, and an image of the light is 
formed on the CCD 203. A VAP 204 refracts the light 
entering the zoom lens groups 201 and thereby can 
move the image formation position of the incident 
light on the CCD 203. 

Actuators 205, 206, 207, and 208 drive their re- 
spective corresponding moving parts, i.e., the VAP 
204, the variator lens group 201b, the aperture stop 
202, and the focusing lens group 201 d via drivers 
205', 206', 207', and 208', respectively. Reference 
numerals 209 to 212 denote encoders. The encoder 
209 outputs a vertical angle 4> of the VAP 204 in the 
form of, e.g., a voltage. The encoders 210 and 212 
output the absolute positions of the variator lens 
group 201 band the focusing lens group 201 d, respec- 
tively, by voltages. The encoder 211 outputs the aper- 
ture amount of the aperture stop 202 and hence the 
Fno. (F-number) of the entire optical system by a vol- 
tage. These output values except the aperture value 
are supplied to a CPU 224 as a lens group control 
means. The aperture value is differentiated by a dif- 
ferentiating circuit 21 3 and supplied to the CPU 224. 

An auto focus (AF) mechanism of the third em- 
bodiment will be described below. Note that in the 
third embodiment, the AF mechanism is described by 
taking so-called TV signal AF as an example, in which 
the focusing lens group is moved such that the fre- 
quency component of a luminance signal of a video 
signal becomes highest. However, the present inven- 
tion can be similarly applied to ohter method, e.g., a 
TCL method or infrared AF. 

An output video signal from the CCD 203 is 
passed through a video signal processor 214 and lim- 
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ited in region by a gate 219. A peak detector 220 de- 
tects a peak value of a luminance signal in a prede- 
termined region and sends the peak value to the CPU 
224 as information S 0 pertaining to the current focus- 
ing state. When an oscillator 223 supplies a driving 
signal of a constant period to the driver 208* via the 
CPU 224, the focusing lens group 201 d is so driven 
as to finely oscillate in the optical axis direction. Con- 
sequently, the output S 0 from the peak detector 220 
also oscillates in synchronism with the oscillation of 
the lens group. This state is shown in Fig. 6A. 

In Fig. 6A, the output S 0 from the peak detector 
220 is plotted on the ordinate, and the position of the 
focusing lens group 20 1d is plotted on the abscissa. 
A position P M at which the peak value of S 0 is obtained 
is an in-focus position. Therefore, if the focusing lens 
group 201 d finely oscillates in an out-of-focus state, 
e.g., when the focusing lens group 201d is located at 
P N or P Ft the phase of the output S 0 with respect to the 
direction of displacement of the focusing lens group 
201d is shifted 180° in accordance with whether the 
focusing lens group 201 d is positioned on the near 
distance side (P N side) or on the far distance side (P F 
side), as indicated by amplitudes A N and A F of the out- 
put S 0 at their respective positions. 

Nearfocus and far focus, therefore, can be deter- 
mined by sending the output from the peak detector 
220 to a phase comparator 222 via a frequency de- 
tector 221 and comparing the output with the phase 
of the reference signal from the oscillator 223. Ampli- 
tudes A of the output are An and A F when the focusing 
lens group 20 1d is positioned on the nearfocus and 
the far focus sides, respectively, and A M = 0 in an in- 
focus state. 

By performing synchronous detection for these 
signals using the output from the oscillator 223 as a 
reference timing, a signal S 1( Fig. 6B f is obtained. 
That is, assuming the signal in the near distance is in 
phase with the reference timing, a positive signal is 
output on the near distance side, and a negative sig- 
nal is output on the far distance side, as the synchron- 
ous detection signal Note that if the circuit config- 
uration is made such that the far distance signal is in 
phase with the reference signal, a negative signal is 
output on the near distance side, and a positive signal 
is output on the far distance side, as the synchronous 
detection signal 

As discussed above, the value of the signal S t is 
"0" in an in-focus state, and its absolute value increas- 
es with the defocus amount. However, the signal 
decreases as the defocus amount increases. There- 
fore, if the signal S 0 becomes smaller than a refer- 
ence level V H , a signal to be used is not the signal 
but a signal S 2 , Fig. 6C, obtained by comparing the 
signal S 0 generated when the focusing lens group 
201d is located in the immediately preceding field 
with the signal S 0 generated at the current position of 
the lens group. Since the signal S 2 shown in Fig. 6C 



is the comparator output, this signal is equivalent to 
a direction signal which indicates the ascending por- 
tion or the descending portion of the peak shown in 
Fig. 6A. That is, the signal indicates the direction in 

5 which the lens group is to be driven with a given value 
either in the positive or negative direction from the in- 
focus position. More specifically, if the defocus 
amount is large (S 0 < V H ), the actuator 208 is driven 
at a fixed speed in accordance with the signal S 2 (a 

10 hill-climbing mode). If the defocus amount is near the 
in-focus position (S 0 ^ Vh), the actuator 208 is driven 
in accordance with the signal S n (a closed loop mode). 
Consequently, an in-focus state can be obtained in 
either case. 

15 Of a series of operations discussed above, the 

fine oscillation of the focusing lens group 201 d must 
be done in synchronism with driving of the CCD 203. 
The CPU 224 controls the operation such thatthe tim- 
ing at which the video signal processor 214 receives 
20 the signal from the CCD 203 and the output timing of 
the oscillator 223 are synchronized with a field pulse 
signal O/E (O: odd field/E: even-field) generated by 
a timing pulse generator 229. 

An operation of increasing resolution by using a 
25 pixel shift according to the third embodiment will be 
described below. In Figs. 5A and 5B f reference nu- 
meral 227 denotes a clock driver for driving the CCD 
203. A delay circuit 228 delays transfer pulse signals 
($1 , $2) from the timing pulse generator 229 by a 1 /2 

30 pixel in accordance with whether the field is an even- 
field or an odd field. A switch 231 is used to send field 
information to the delay circuit 228. The switch 231 is 
interlocked with a switch 230 for turning on and off a 
periodical pixel shifting operation. The switches 230 

35 and 231 are turned on or off together. In performing 
the pixel shift, these switches 230 and 231 are turned 
on, and the delay circuit 228 checks whether the field 
pulse signal (O/E) from the timing pulse generator 
229 indicates an even-field or an odd field. If the tim- 

40 ing pulse signal is an even-field signal, the delay cir- 
cuit 228 delays the transfer pulse signals ($1 , $2) for 
a read horizontal shift register by a 1/2 pixel. If the 
timing pulse signal is an odd field signal, the delay cir- 
cuit 228 sends the transfer pulse signals directly to 

45 the CCD driver 227 without delaying the signals. Al- 
ternatively, the delay circuit 228 delays the transfer 
pulse signals (+1, $2) for the read horizontal shift reg- 
ister by a 1/2 pixel if the field pulse signal is an odd 
field signal, and does not delay the transfer pulse sig- 

50 nals if the field pulse signal is an even-field signal. 
The CCD 203 is driven in synchronism with the pulse 
signals (4>1, $2) from the timing pulse generator 229. 
The VAP 204 is also driven such that the image for- 
mation position is horizontally shifted a 1/2 pixel on 

55 the image sensing plane for each field. Consequent- 
ly, it is possible to obtain resolution equivalent to that 
obtained when the number of pixels in the horizontal 
direction is nearly doubled. 



8 



15 



EP 0 669 757 A2 



16 



On the other hand, if the switches 230 and 231 
are OFF, the field pulse signal (O/E) is not supplied to 
the delay circuit 228. Therefore, the transfer pulse 
signals (<t>1, <j>2) are constantly supplied to the CCD 
driver 227 after being delayed by a 1/2 pixel, and the 
periodical driving of the VAP 204 is stopped. That is, 
ON/OFF of the switches 230 and 231 corresponds to 
ON/OFF of the pixel shift mode. 

When the switches 230 and 231 are ON, the tim- : 
ing the signal read out from the CCD 203 is shifted 
by a 1/2 pixel in accordance with whether the field is 
an odd field or an even-field. Therefore, by outputting 
this signal as a composite signal via a camera encod- 
er 232, on the receiving apparatus side a high- 
resolution image can be obtained on, e.g., a high- 
resolution monitor from signals of both the fields. 

Note that in Figs. 5A and 5B, a user can visually 
check the output from the camera encoder 232 
through an EVF (Electric View Finder) 233. The reso- 
lution of the EVF 233 can be relatively low when its 
performance is taken into account However, in the 
high-resolution mode, i.e., when the switches 230 and 
231 are ON, characters such as "HD" are superim- 
posed on the display screen of the EVF 233. This al- 
lows a user to confirm in the low resolution EVF 233 
that photographing is currently being done in the high- 
resolution mode. 

Note that in the third embodiment, the read drive 
timing of the CCD 203 is shifted a 1/2 pixel in even 
and odd fields. However, it is also possible to write 
even-field signals in every other pixels of a memory 
having the number of horizontal pixels which is twice 
the number of pixels of the CCD 203, and to write odd 
field signals between the even-field signals. In this 
case the signals are sequentially read out pixel-by- 
pixel in the horizontal direction from the memory. Al- 
ternatively, a first field signal can be temporarily stor- 
ed in a memory having the same number of pixels as 
that of the CCD 203. In this case, in reading out a sec- 
ond-field signal the signal from the memory and the 
output signal from the CCD 203 are alternately output 
pixel-by-pixel in the horizontal direction. 

Automatic exposure control in the third embodi- 
ment will be described below. 

In Figs. 5 A and 5B, an optical image of an object 
to be photographed taken by the taking lens groups 
201a to 201 d is formed on the image sensing plane 
of the CCD 203 after the light quantity of the image 
is adjusted by the aperture stop 202. The output video 
signal from the CCD 203 is limited in region by a gate 
215 and subjected to integrating detection performed 
by an integrating detector 216. The resulting signal is 
output to an operational amplifier 21 7 as the intensity 
of luminance. The operational amplifier 217 com- 
pares this voltage value with the voltage of a refer- 
ence voltage generator 218, amplifies the resultant 
difference signal, and outputs the signal to the CPU 
224 as a difference signal D 2 . The CPU 224 supplies 



a signal to the driver 207' such that this difference 
voltage becomes "0", thereby driving the aperture ac- 
tuator 207. In order that the aperture stop 202 is 
opened/closed stably, the CPU 224 performs control 

5 such that a differential output D t indicative of the 
aperture position from the differentiating circuit 213 
is fed back and so the level of D 1 is held at a prede- 
termined value. With the above operation, exposure 
is automatically optimized in the third embodiment 

10 Therefore, whether the automatic exposure control is 
completed can be determined by checking whether 
the output D, from the differentiating circuit 213 is 
close to "0" or whether the difference signal D 2 from 
the operational amplifier 217 is dose to "0". 

15 A zoom function in the third embodiment will be 

described below. 

Referring to Figs. 5Aand 5B, when a zoom switch 
226 is turned on a direction signal corresponding to 
the depressed direction (T side/W side) of the zoom 

20 switch 226 is supplied to the CPU 224. Note that the 
zoom switch 226 designates execution of close up 
photographing in a tele mode when depressed to the 
T side and execution of wide-angle photographing in 
a wide mode when depressed to the W side. Upon re- 

25 ceiving the direction signal, the CPU 224 sends a sig- 
nal to the driver 206' to move the variator lens group 
201 b in a predetermined direction. The optical system 
used in the third embodiment is a so-called variable 
focal lens group in which the focusing lens group 201 d 

30 is closer to the image plane than the variator lens 
group 201 b. Consequently, the in-focus plane moves 
with movement of the variator lens group 201b. 
Therefore, the CPU 224 drives the variator lens group 
201b and at the same time must also drive the focus- 

35 Ing lens group 201 d on the basis of data from a ROM 
225 in orderto keep the in-focus state of the CCD 203. 
Note that if the optical system is not a variable focal 
lens group, the focusing lens group can be kept f ixed. 
Whether the zoom operation is completed can be de- 

40 termined by checking ON/OFF of the signal from the 
zoom switch 226. 

An AF operation performed by the CPU 224 will 
be described below with reference to Fig. 7. 

Referring to Fig. 7, in step S10 the CPU 224 

45 starts the AF operation. Thereafter, the CPU 224 
turns off the switches 230 and 231 in step S11 and 
also turns off the display of characters "HD* indicat- 
ing the high-resolution mode from the image in the 
EVF 233 in step S12. In step S13, the CPU 224 

so checks whether S 0 > V H as described above in Fig. 
6A. If S 0 ^ V H , the flow advances to step S14 because 
the defocus amount is large. In step S14, the CPU 224 
checks whether S 2 > 0. If S 2 > 0, the CPU 224 moves 
the focusing lens group 201 d toward the far distance 

55 side in step S15. If S 2 ^ 0, the CPU 224 moves the 
focusing lens group 201 d toward the near distance 
side in step S16. The CPU 224 repeatedly executes 
this processing until S 0 ^ V H is obtained in step S13. 
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If S 0 ^ V H in step S13, the flow advances to step 
S17, and the CPU 224 checks whether |s, I > V S1 . 
V S i is a predetermined value close to "0". If | I > 
V S i, the CPU 224 so drives the focusing lens group 
201dthat | Si | = 0 in stepS18. 

If | Si | =S V S1 in step S17, the CPU 224 determi- 
nes that an in-focus state is obtained. Therefore, in 
step S19 the CPU 224 stops the focusing lens group 
* 201 d. 

Thereafter, the flow advances to step S20, and 
the CPU 224 checks whether > V D1 is established 
for the differential output D, which is supplied from 
the differentiating circuit 213 and indicates the aper- 
ture position of the aperture stop 202. V D1 is a prede- 
termined value close to "0". If D A > V D1 , in step S21 the 
CPU 224 checks whether D 2 > V D2 is established for 
the difference signal D 2 from the operational amplifi- 
er 217. V D2 is also a predetermined value close to "0". 
If D 2 > V D2 , in step S22 the CPU 224 performs the au- 
tomatic exposure control discussed above, and the 
flow advances to step S26. 

If D, ^ V D1 in step S20 or D 2 ^ V D2 in step S21, 
the flow advances to step S23, and the CPU 224 
checks whether the zoom switch 226 is ON, i.e., is de- 
pressed. If the zoom switch 226 is OFF, the flow ad- 
vances to step S24, and the CPU 224 turns on the 
switches 230 and 231. In step S25, the CPU 224 su- 
perimposes the characters "HD" on the EVF 233. This 
display is not limited to "HD" but can be an icon or 
some other characters. If the zoom switch 226 is ON 
in step S23, the flow advances to step S26. 

In step S26, the CPU 224 checks whether | S, | 
> V s2 . V S2 is a predetermined value meeting V S2 > V S1 . 
If I Si | >V S2 in step S26 t the flow returns to step S11; 
otherwise, the flow returns to step S1 9. Since the re- 
lation V s2 > V S1 is established in this manner, no hunt- 
ing occurs in the AF processing even if the level of 
is unstable. 

The AF processing is performed as described 
above. In the third embodiment, therefore, the 
switches 230 and 231 are turned on and the display 
"HD M is superimposed only when all of the three con- 
ditions, (1) I S t I < V sll (2) D, ^ V D1 or D 2 £ V D2 , and 
(3) the switch 226 is OFF, are satisfied. That is, pho- 
tographing is done in the high-resolution mode only 
when AF is near an in-focus state, exposure is nearly 
appropriate, and there is no zoom command. There- 
fore, in the third embodiment an unnecessary pixel 
shift is not performed in an out-of-focus state, when 
the entire image is black or white on the screen, or 
during zooming. 

In the third embodiment, the pixel shift and the 
superimpose display are performed when all of the 
above conditions (1 ) to (3) are met, and so the method 
can be considered to be most efficient However, the 
pixel shift and the superimpose display can also be 
done when one or two of the conditions (1) to (3) are 
met. Even in this case, a power-saving effect can be 



obtained to some extent 

According to the third embodiment as discussed 
above, whether the pixel shift is to be performed is 
determined in accordance with the. input signal to the 

5 CPU for controlling the image sensing apparatus. 
This prevents interference between the pixel shift 
and other automatic adjusting functions of the image 
sensing apparatus. In addition, an unnecessary pixel 
shift is not performed in an out-of-focus state or when 

w exposure is inadequate. This effectively saves the 
consumption power. Consequently, it is possible to 
constantly obtain high-resolution images with a high 
efficiency. 

15 <4th Embodiment 

The fourth embodiment according to the present 
invention will be described below. 

Fig. 8 is a block diagram showing the arrange- 

20 ment of an image sensing apparatus of the fourth em- 
bodiment Reference numeral 301 denotes a record- 
ing system including CCDs. Atrichromatic separation 
prism 310 separates light from an object to be photo- 
graphed into three colors, red (R), green (G), and blue 

25 (B), and supplies these three light components to 
three CCDs 311a to 311c. R, G, and B filters are ad- 
hered to the CCDs 311a, 311b, and 311c, respective- 
ly. In the fourth embodiment, the pixel arrangements 
in the three CCDs 31 1 a to 31 1 c are precisely identical. 

30 That is, no pixel shift is, done between the CCDs, so 
an image of an object to be photographed is projected 
at the same position in each CCD. Reference numeral 
312 denotes an A/D converter for converting outputs 
from the CCDs 311a to 311c into digital signals; and 

35 313, a memory. The RGB image data obtained by the 
CCDs 311a to 311c are recorded in a recording unit 
314, such as a semiconductor memory card or a hard 
disk, via the A/D converter 312 and the memory 313. 
This recording operation is controlled by a CPU 315. 

40 An image sensing system 302 controls exposure, 

focus detection, zooming, and the like processing. A 
VAP 303 is a pixel shifting means for changing the rel- 
ative position of an image of an object with respect to 
each CCD. As explained in Japanese Patent Laid- 

45 Open No. 2-124518, the VAP 303 is an optical ele- 
ment formed by sealing a transparent elastic body be- 
tween parallel plates such as glass plates. The optical 
performance of a transmitted light beam is arbitrarily 
changed by changing the angles of these parallel 

so plates using an external biasing force. Examples of 
the transparent elastic body sandwiched between the 
parallel plates are water, alcohol, glycol, silicone oil, 
silicone gel, silicone rubber, and organic oil. 

A motion detector 331 detects the amount of a 

55 camera shake in real time by using an acceleration 
sensor. In accordance with the detected amount, the 
front and rear glass plates of the VAP 303 are moved 
by a driving mechanism (not shown), thereby chang- 
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ing the vertical angle of the VAP 303. In this manner, 
a blur of an image caused by the camera shake is cor- 
rected. A drive controller 330 performs drive control, 
such as focus detection and exposure of the image 
sensing system 302, and drive control of the driving 5 
mechanism for the glass plates of the VAP 303. The 
drive timing of the drive controller 330 is controlled by 
the CPU 315. 

A high-frequency component detector 304 ex- ' 
tracts a high-frequency component from data in a 10 
predetermined area of the memory 313. The CPU 31 5 
and the drive controller 330 keep driving the image 
sensing system 302 until the high-frequency compo- 
nent is maximized, i.e., an in-focus state is obtained. 
The focus detection method in the fourth embodiment 15 
is a so-called "hill-climbing servo method" which is 
well-known in the field of a video camera and in which 
an image sensing optical system is so controlled that 
a high-frequency component of an output signal from 
a CCD is maximized. This method is disclosed in Ish- 20 
ida et a!., "Automatic Focus Adjustment of Television 
Camera According to Hill-climbing Servo Method", 
NHK Technology, Vol. 17, No. 1, and so a detailed de- 
scription thereof will be omitted. 

In the fourth embodiment, a pixel shift is per- 25 
formed by changing the relative position of an image 
of an object with respect to each CCD by further 
slightly changing the vertical angle of the VAP 303. If 
the image sensing apparatus vibrates during photo- 
graphing, the influence of the vibration can be re- 30 
duced by superposing the driving of the pixel shift on 
the driving of the VAP 303 for vibration isolation. Fig. 
9 is a timing chart showing the state in which signals 
exposed and stored in the CCDs are sequentially 
read out while the VAP 303 is performing the pixel 35 
shift Since the operations of the CCDs 311a to 311c 
are exactly the same, the operation of the CCD 311b 
will be described as an example. 

Referring to Fig. 9, A1 represents the first expos- 
ure, and the read of a signal stored by the exposure 40 
A1 is represented by C1 . That is, after the exposure 
A1 is completed, the read C1 of the stored signal is 
started, and at the same time the VAP 303 is driven 
to perform a pixel shift. Likewise, signals stored in ex- 
posures A2, A3, and A4 are read out in C2, C3, and 45 
C4, respectively. After each of the exposures A2, A3, 
and A4 is completed, the VAP 303 is driven to perform 
a pixel shift. 

The pixel shift in the fourth embodiment will be 
described below. Fig. 10 shows the way the position so 
of an image of an object to be photographed is shifted 
on the CCD by the VAP 303. Reference symbols 1a t 
1 b, 1 c, 2a, 2b, and 2c denote some pixels on the CCD 
311b; and p, a pixel pitch. In the fourth embodiment, 
it is assumed that the pixel pitches are identical in the 55 
vertical and horizontal directions. Note that since the 
pixel arrangements of the CCDs 311a and 311c are 
identical with that of the CCD 311b, only the CCD 



311b will be described below. 

After an image of an object is photographed once 
at the initial position, the VAP 303 is driven such that 
images of the object projected on the pixels 1a, 1b f 
1 c, 2a, 2b, and 2c are shifted by a p/2 pitch to the right 
in Fig. 10. Thereafter, photographing is executed 
again. The driving amount at this time is calculated as 
follows. 

For the simplicity of explanation, the positional 
relationship between the VAP 303 and the image 
sensing system 302 is extracted from Fig. 8 and illu- 
strated in Fig. 11. In Fig. 11 f symbols have the follow- 
ing meanings. 

8: The angle at which light rays are inclined by 
the VAP 303. 

a: The distance from the VAP 303 to an image 
pick up plane 352. 

d: The shift amount (pixel shift amount) of an 
image of an object on the image sensing plane 352. 
In this case, since d is sufficiently small, the following 
equation is established: 

tand = d/a 

Since a » d, 8 = d/a is established. 

In order for the pixel shift amount d to be p/2, 
therefore, it is only necessary to drive the VAP 303 un- 
til the inclination angle 6, with which 8 = d/a = p/2a is 
established, is obtained. Generally, control of the 
VAP 303 is done by the angle defined, between the 
front and rear glass plates constituting the VAP 303, 
i.e., the vertical angle, rather than by the inclination 
angle 8 of light rays. Calculations for this control will 
be discussed in detail in the fifth embodiment to be 
described later. 

In Fig. 10, the images of the object are photo- 
graphed as they are shifted in units of p/2 downward 
and to the left in the same manner as mentioned 
above. Thereafter, the images are returned to the ini- 
tial positions. The data obtained by projecting colors 
R, G, and B while the object images are shifted four 
times in this way are rearranged in the memory 313. 
This makes it possible to reconstruct, for each of R, 
G, and B, a high-resolution image having. the number 
of pixels which is four times the number of pixels of 
each CCD. 

The high-frequency component detector 304 de- 
tects a high-frequency component from the images 
obtained as discussed above, e.g., from an image of 
G component, thereby performing focus detection. 
Since the pixel shift is done for the image information 
to be detected, the information amount is doubled in 
each of the vertical and horizontal directions. There- 
fore, a larger amount of a high-frequency component 
is extracted than before the pixel shift is performed. 
The image sensing system 302 is kept driven until the 
high-frequency component is maximized, i.e., an in- 
focus state is obtained. 

According to the fourth embodiment as dis- 
cussed above, the image sensing apparatus includes 
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the means for photographing an image of an object to 
be photographed by using n (n is 2 or larger) image 
sensors, and the means for exposing the image of the 
object a plurality oF-number of times while changing 
the relative position of the image to each image sen- 
sor. As a result, it is possible to obtain high-resolution 
images. This makes detection of a larger amount of a 
high-frequency component and therefore a more ac- 
curate focus detection feasible. 

<5th Embodiment 

The fifth embodiment of the present invention 
will be described below. 

An image sensing apparatus of the fifth embodi- 
ment has an arrangement identical with that of the 
fourth embodiment shown in Fig. 8, so a detailed de- 
scription thereof will be omitted. In the fifth embodi- 
ment, of the three CCDs 311a to 311c illustrated in 
Fig. 8, the CCDs 311a and 311b have a filter of G, and 
a pixel shift is done between them. Note that the CCD 
311c has stripe filters of R and B. The arrangement 
of the fifth embodiment is similar to that disclosed in 
"Highvision 2/3-inch compact CCD camera using dual 
green method" in Television Society Magazine, "im- 
age Information Engineering and Broadcasting Tech- 
nology", Vol. 47, No. 2, 1993 described in "BACK- 
GROUND OF THE INVENTION". Also, a method of 
producing RGB signals for all pixels from photograph- 
ed data is known to those skilled in the art. Therefore, 
a method of a pixel shift with respect to a luminance 
signal of an image of an object to be photographed 
will be described below. 

Referring to Fig. 12, reference symbols 31a, 31b, 
31c, 32a t 32b, and 32c denote some pixels in the CCD 
311a. In the CCD 311b, on the other hand, pixels are 
arranged at positions corresponding to positions be- 
tween the pixels in the CCD 311a, i.e., arranged at 
pixel-shifted positions, as indicated by 31 a\ 31b', 
31c*, 32a', 32b', and 32c\ After photographing is per- 
formed in this state, the relative positions of images 
of an object to be photographed are shifted in the di- 
rection indicated by arrows in Fig. 12 by a VAP 303, 
and again photographing is performed. In this meth- 
od, a pixel arrangement of a G component as a lumi- 
nance component is originally shifted. Therefore, for 
a luminance component it is possible to obtain the 
number of pixels which is equivalent to the number of 
pixels in the RGB three sensor method described in 
the fourth embodiment by performing photographing 
a total of two times. 

Note that in the fifth embodiment, in two of the 
three CCDs the pixel shift is done in both the horizon- 
tal and vertical directions. However, the pixel shift 
can also be performed only in one direction. 

In addition, if n (2 ^ n) CCDs are employed and 
m (2 ^ m ^ n) CCDs having filters of the same color 
are pixel-shifted by 1/m each in the horizontal orvert- 



ical direction, ((k + 1) x m) pixel data can be obtained 
for a luminance component by pixel shifting an image 
of an object to be photographed to a position at which 
no pixel is present k times by using a VAP or the like. 
5 These data are rearranged in a memory 313. This 
makes it possible to reconstruct, for a luminance com- 
ponent a high-resolution image having a data 
amount (k + 1) times as large as the data amount of 
each CCD. 

10 According to the fifth embodiment as discussed 
above, it is possible to synthesize images with the 
same high-resolution as in the fourth embodiment 
within a shorter time than in the fourth embodiment. 

15 <6th Embodiment 

The sixth embodiment according to the present 
invention will be described below. 

Conventionally, a method has been proposed by 
20 which an image of an object to be photographed is 
divided into a plurality of regions and photographed, 
and the photographed images are synthesized to ob- 
tain a high-resolution image. The sixth embodiment is 
characterized by further increasing the resolution in 

25 this method. 

The arrangement of an image sensing apparatus 
of the sixth embodiment is nearly identical with that 
of the fourth embodiment shown in Fig. 8 except that 
an optical system for forming an image of an object 
30 to be photographed on each CCD is different from 
that of the fourth embodiment. Fig. 13 shows an opt- 
ical system in the sixth embodiment, in which the 
same reference numerals as in Fig. 8 denote the 
same parts and a detailed description thereof will be 
35 omitted. 

Referring to Fig. 13, dotted lines 351 represent a 
light beam from an object to be photographed. An im- 
age of the beam 351 is first formed on a primary im- 
age formation plane 352 via a VAP 303 and an image 
40 sensing system 302. The image of the object once 
formed on the primary image formation plane 352 is 
divided by a secondary image formation optical sys- 
tem 353 and formed as secondary images on CCDs 
354a and 354b. In this case the object images formed 
45 as the secondary images on the CCDs 354a and 
354b are essentially pixel-shifted by the VAP 303. 
The images formed on the two CCDs 354a and 354b 
are synthesized in a memory 313 to yield an image 
having the number of pixels which is twice the number 
so of pixels of one CCD. 

According to the sixth embodiment as discussed 
above, images with a higher resolution can be photo- 
graphed. 

In addition, in the sixth embodiment a still higher 
55 resolution can be achieved by periodically driving the 
VAP 303 and further performing a pixel shift as illu- 
strated in Fig. 9 of the fourth embodiment, thereby 
performing photographing a plurality of times. 

12 
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In each of the first to sixth embodiments descri- 
bed above, a VAP is used as the pixel shifting means. 
However, the present invention is not limited to these 
embodiments and can be realized by using the follow- 
ing means as well. Figs. 14Ato 14E illustrate means 5 
for changing the relative position of an image of an ob- 
ject to be photographed with respect to a CCD, i.e., 
pixel shifting means, including a VAP used in the 
above embodiments. 

Fig. 14A shows the VAP method discussed 10 
above. Assuming that the vertical angle is <|> ( the in- 
clination angle of a light beam is d, and the refractive 
index of a VAP is n, 8 a (n - 1) x <j> is established. 

Fig. 14B shows a reflection argument method. 
When a mirror surface reflecting plate is inclined by 15 
9, 5 = 2 x 9 is established for an inclination angle 6 of 
a light beam. Note that in the pixel shifting methods 
in Figs. 14Aand 14B, § and 9 are controlled by per- 
forming calculations such that an image of an object 
to be photographed is shifted a predetermined 20 
amount on an image sensing plane. 

Fig. 14C shows a rotary prism method. When a 
prism with a width L is inclined by <j>, a light beam is 
shifted parallel by d which is given by d = ((n - 1) x L 
x <t>)/n. 25 

Fig. 14D illustrates a moving prism method. 
When two prisms each having a vertical angle a and 
a refractive index n are arranged in the reverse direc- 
tions as in Fig. 14D, the relationship between a dis- 
tance D between the prisms and a distance d by which 30 
a light beam is shifted is given by d s (n - 1) x a x D. 
Therefore, an image can be shifted by changing D. In 
the pixel shifting methods illustrated in Figs. 14C and 
14D, (pixel shift amount) = d when the prism or the 
prisms are arranged immediately before the CCD. If 35 
another optical system is inserted between the prism 
or the prisms and the CCD, the driving amount d need 
only be determined by taking account of the effect of 
that optical system. 

Fig. 14E illustrates a moving lens group method. 40 
A lens group with a focal length f arranged in a taking 
lens group system is vertically shifted by D with re- 
spect to the optical axis. Assuming the distance be- 
tween the lens group and the in-focus plane is I, a 
shifted amount d of an image on the in-focus plane is 45 
represented by d =s I x D/f . 

As discussed above, the VAP method, Fig. 14A, 
as explained in the fifth embodiment is merely one 
example of the pixel shifting methods, so the means 
of dynamically shifting pixels can be achieved by va- so 
rious methods. 

Among other methods by which similar effects 
can be obtained are a method as disclosed in Japa- 
nese Patent Publication No. 57-31701 , in which a po- 
larizer capable controlling the direction of polarized 55 
light is arranged in an optical path; a method as dis- 
closed in Japanese Patent Laid-Open No. 3-201883, 
in which an entire image sensing apparatus is dis- 



placed; a method in which an image is obtained by 
displacing an image sensing optical system either 
partially or entirely to change the relative positional 
relationship between the image formed by an input 
optical system and a light-receiving portion of a CCD 
with time; and a method in which a CCD itself is vi- 
brated on a plane normal to the optical axis by using 
a piezoelectric element or the like. 

As many apparently widely different embodi- 
ments of the present invention can be made without 
departing from the spirit and scope thereof, it is to be 
understood that the invention is not limited to the spe- 
cific embodiments thereof except as defined in the 
appended claims. 

Claims 

1. An image sensing apparatus comprising: 

image forming means for forming an im- 
age of an object to be photographed; 

image sensing means for photographing 
the image of the object formed by said image 
forming means; 

displacing means for displacing the image 
of the object formed by said image forming 
means and said image sensing means relative to 
each other; and 

control means for performing switching 
between a first image sensing mode in which 
said displacing means performs a vibration cor- 
rection to yield a low resolution image and a sec- 
ond image sensing mode in which said displacing 
means performs a pixel shift to yield a high- 
resolution image. 

2. The apparatus according to claim 1 , wherein 

said image sensing means consists of a 
plurality of image sensors each corresponding to 
one pixel, and 

in the second image sensing mode the im- 
age of the object is shifted by 1/N (N is a natural 
number of not less than 2) of a pixel pitch of said 
image sensing means by said displacing means, 
and resulting N images of the object are photo- 
graphed and synthesized to yield a high- 
resolution image. 

3. The apparatus according to claim 2, further com- 
prising a memory for synthesizing the N images 
of the object obtained in the second image sens- 
ing mode. 

4. The apparatus according to claim 2, further com- 
prising vibration detecting means for detecting a 
vibration of said image sensing apparatus, 

wherein in the first image sensing mode 
said displacing means is controlled in accor- 
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dance with an amount of the vibration detected by 
said vibration detecting means. 

5. The apparatus according to claim 1, further com- 
prising motion detecting means for detecting a 
motion vector of the object, 

wherein said control means performs 
switching between the first and second image 
sensing modes in accordance with information on 
the motion vector of the object detected by said 
motion detecting means. 

6. The apparatus according to claim 1, wherein said 
displacing means includes a variable angle prism 
and an actuator for controlling a vertical angle of 
said variable angle prism. 

7. The apparatus according to claim 2, wherein said 
image sensor is a CCD. 

8. The apparatus according to claim 1 , wherein said 
displacing means changes an argument of a mir- 
ror surface reflecting plate. 

9. The apparatus according to claim 1 , wherein said 
displacing means rotates a prism. 

10. The apparatus according to claim 1 , wherein said 
displacing means changes relative positions of a 
plurality of prisms. 

11. The apparatus according to claim 1 , wherein said 
displacing means changes a position of a lens 
group with respect to an optical axis. 

12. An image sensing apparatus comprising: 

image forming means for forming an im- 
age of an object to be photographed; 

image sensing means for photographing 
the image of the object formed by said image 
forming means; 

displacing means for displacing the image 
of the object formed by said image forming 
means and said image sensing means relative to 
each other; 

synthesizing means for obtaining a high- 
resolution image by synthesizing a plurality of im- 
ages of the object obtained by performing a pixel 
shift by said displacing means; 

changing means for changing photograph- 
ing conditions; and 

control means for determining, in accor- 
dance with the photographing conditions 
changed by said changing means, whether said 
displacing means is to be driven. 

13. The apparatus according to claim 12, wherein 
said changing means includes automatic 



focusing means for obtaining an in-fbcus state by 
automatically controlling said image forming 
means in accordance with the image of the ob- 
ject, and 

s said control means drives said displacing 

means if the in-focus state is obtained by said au- 
tomatic focusing means and does not drive said 
displacing means if no in-focus state is obtained. 

10 14. The apparatus according to claim 12, wherein 
said changing means includes automatic 
exposing means for obtaining a proper exposure 
by automatically controlling said image forming 
means in accordance with the image of the ob- 
is ject, and 

said control means drives said displacing 
means if the proper exposure is obtained by said 
automatic exposing means and does not drive 
said displacing means if no proper exposure is 
20 obtained. 

15. The apparatus according to claim 12, wherein 

said changing means includes zooming 
means for controlling a photographing magnifica- 
25 tion of the image of the object, and 

said control means drives said displacing 
means if said zooming means is not being driven 
and does not drive said displacing means if said 
zooming means is being driven. 

30 

16. The apparatus according to claim 12, wherein 
said displacing means includes a variable angle 
prism and an actuatorfor controlling a vertical an- 
gle of said variable angle prism. 

35 

17. The apparatus according to claim 12, further 
comprising viewing means through which the im- 
age of the object can be viewed, 

wherein when said control means drives 
40 said displacing means, information indicating the 

driving is sent to said viewing means. 

18. The apparatus according to claim 12, wherein 
said displacing means changes an argument of a 

45 mirror surface reflecting plate. 

19. The apparatus according to claim 12, wherein 
said displacing means rotates a prism. 

so 20. The apparatus according to claim 12, wherein 
said displacing means changes relative positions 
of a plurality of prisms. 

21. The apparatus according to claim 12, wherein 
55 said displacing means changes a position of a 

lens group with respect to an optical axis. 

22. An image sensing apparatus comprising: 

14 
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image forming means for forming an im- 
age of an object to be photographed; 

image sensing means for photographing 
the image of the object formed by said image 
forming means; 

displacing means for displacing the image 
of the object formed by said image forming 
means and said image sensing means relative to 
each other; 

synthesizing means for obtaining a high- 
resolution image by synthesizing a plurality of im- 
ages of the object obtained by performing a pixel 
shift by said displacing means; and 

vibration correcting means for selectively 
performing a vibration correction by using said 
displacing means. 

23. The apparatus according to claim 22, wherein 
said image sensing means consists of a 
plurality of image sensors each corresponding to 
one pixel, 

said displacing means displaces the im- 
age of the object by 1/N (N is a natural number of 
not less than 2) of a pixel pitch of said image 
sensing means, and 

said synthesizing means synthesizes re- 
sulting N images of the object to yield a high- 
resolution image. 



said displacing means rotates a prism. 

29. The apparatus according to claim 22, wherein 
said displacing means changes relative positions 

5 of a plurality of prisms. 

30. The apparatus according to claim 22, wherein 
said displacing means changes a position of a 
lens group with respect to an optical axis. 

10 

31. An imaging apparatus comprising means for 
forming an image of an object, means for sensing 
the image and means for effecting relative move- 
ment between the forming and sensing means to 

15 enable motion compensation and/or to enable 
formation of a high resolution sensed image. 

32. An imaging apparatus according to claim 31, 
which further comprises means for combining a 

20 number of images sensed by the sensing means. 
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24. The apparatus according to claim 23, wherein 30 
said image forming means separates the 

image of the object to form a plurality of images, 
the same number of said image sensing 

means as the number of the images formed by 

said image forming means are provided, and 35 
in at least two of said plurality of image 

sensing means arrangements of image sensors 

are pixel-shifted from each other. 



25. The apparatus according to claim 22, wherein 40 

said image forming means forms at least 
two secondary images of the object, 

said displacing means displaces the at 
least two secondary images relative to each 
other, and 45 

said image sensing means photographs 
the at least two secondary images. 

26. The apparatus according to claim 22, wherein 

said displacing means includes a variable angle 50 
prism and an actuator for controlling a vertical an- 
gle of said variable angle prism. 

27. The apparatus according to claim 22, wherein 
said displacing means changes an argument of a 55 
mirror surface reflecting plate. 



28. The apparatus according to claim 22, wherein 
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